Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters.

BACKGROUND: Artificial intelligence (AI) is becoming increasingly used in medical education, but our understanding of the validity of AI-based assessments (AIBA) as compared with traditional clinical expert-based assessments (EBA) is limited. In this study, the authors aimed to compare and contrast the validity evidence for the assessment of a complex clinical skill based on scores generated from an AI and trained clinical experts, respectively. METHODS: The study was conducted between September 2020 to October 2022. The authors used Kane's validity framework to prioritise and organise their evidence according to the four inferences: scoring, generalisation, extrapolation and implications. The context of the study was chorionic villus sampling performed within the simulated setting. AIBA and EBA were used to evaluate performances of experts, intermediates and novice based on video recordings. The clinical experts used a scoring instrument developed in a previous international consensus study. The AI used convolutional neural networks for capturing features on video recordings, motion tracking and eye movements to arrive at a final composite score. RESULTS: A total of 45 individuals participated in the study (22 novices, 12 intermediates and 11 experts). The authors demonstrated validity evidence for scoring, generalisation, extrapolation and implications for both EBA and AIBA. The plausibility of assumptions related to scoring, evidence of reproducibility and relation to different training levels was examined. Issues relating to construct underrepresentation, lack of explainability, and threats to robustness were identified as potential weak links in the AIBA validity argument compared with the EBA validity argument. CONCLUSION: There were weak links in the use of AIBA compared with EBA, mainly in their representation of the underlying construct but also regarding their explainability and ability to transfer to other datasets. However, combining AI and clinical expert-based assessments may offer complementary benefits, which is a promising subject for future research.

Building low-cost simulators for invasive ultrasound-guided procedures using the V-model.

The use of medical simulators for training technical and diagnostic skills has rapidly increased over the past decade. Yet, most available medical simulators have not been developed based on a structured evaluation of their intended uses but rather out of expected commercial value. Moreover, educators often struggle to access simulators because of cost or because no simulators have been developed for a particular procedure. In this report, we introduce "the V-model" as a conceptual framework to illustrate how simulator development can be guided by the intended uses in an iterative fashion. Applying a needs-based conceptual framework when developing simulators is important to increase the accessibility and sustainability of simulation-based medical education. It will minimize the developmental barriers and costs, while at the same time improving educational outcomes. Two new simulators for invasive ultrasound-guided procedures are used as examples, the chorionic villus sampling model and the ultrasound-guided aspiration trainer. Our conceptual framework and the use cases can serve as a template for future simulator development and documentation hereof.

Does group size matter during collaborative skills learning? A randomised study.

BACKGROUND: Collaborative skills learning in the form of dyad learning compared with individual learning has been shown to lead to non-inferior skills retention and transfer. However, we have limited knowledge on which learning activities improve collaborative skills training and how the number of collaborators may impact skills transfer. We explored the effects of skills training individually, in dyads, triads or tetrads on learning activities during training and on subsequent skills transfer. METHODS: In a randomised, controlled study, participants completed a pre-post-transfer-test set-up in groups of one to four. Participants completed 2 hours of obstetric ultrasound training. In the dyad, triad and tetrad group participants took turns actively handling the ultrasound probe. All performances were rated by two blinded experts using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale and a Global Rating Scale (GRS). All training was video recorded, and learning activities were analysed using the Interactive-Constructive-Active-Passive (ICAP) framework. RESULTS: One hundred one participants completed the simulation-based training, and ninety-seven completed the transfer test. Performance scores improved significantly from pre- to post-test for all groups (p < 0.001, ηp2 = 0.55). However, group size did not affect transfer test performance on OSAUS scores (p = 0.13, ηp2 = 0.06) or GRS scores (p = 0.23, ηp2 = 0.05). ICAP analyses of training activities showed that time spent on non-learning and passive learning activities increased with group size (p < 0.001, ηp2 = 0.31), whereas time spent on constructive and interactive learning activities was constant between groups compared with singles (p < 0.001, ηp2 = 0.72). CONCLUSION: Collaborative skills learning in groups of up to four did not impair skills transfer despite less hands-on time. This may be explained by a compensatory shift towards constructive and interactive learning activities that outweigh the effect of shorter hands-on time.

Consensus on Training and Assessment of Competence in Performing Chorionic Villus Sampling and Amniocentesis: An International Delphi Survey.

INTRODUCTION: The aim of this study was to obtain expert consensus on the content of a curriculum for learning chorionic villus sampling (CVS) and amniocentesis (AC) and the items of an assessment tool to evaluate CVS and AC competence. METHODS: We used a 3-round iterative Delphi process. A steering committee supervised all processes. Seven international collaborators were identified to expand the breadth of the study internationally. The collaborators invited fetal medicine experts to participate as panelists. In the first round, the panelists suggested content for a CVS/AC curriculum and an assessment tool. The steering committee organized and condensed the suggested items and presented them to the panelists in round 2. In the second round, the panelists rated and commented on the suggested items. The results were processed by the steering committee and presented to the panelists in the third round, where final consensus was obtained. Consensus was defined as support by more than 80% of the panelists for an item. RESULTS: Eighty-six experts agreed to participate in the study. The panelists represented 16 countries across 4 continents. The final list of curricular content included 12 theoretical and practical items. The final assessment tool included 11 items, systematically divided into 5 categories: pre-procedure, procedure, post-procedure, nontechnical skills, and overall performance. These items were provided with behavioral scale anchors to rate performance, and an entrustment scale was used for the final overall assessment. CONCLUSION: We established consensus among international fetal medicine experts on content to be included in a CVS/AC curriculum and on an assessment tool to evaluate CVS/AC skills. These results are important to help transition current training and assessment methods from a time- and volume-based approach to a competency-based approach which is a key step in improving patient safety and outcomes for the 2 most common invasive procedures in fetal medicine.

Plasma progesterone, estradiol, and unconjugated estriol concentrations in twin pregnancies: Relation with cervical length and preterm delivery.

INTRODUCTION: The aim of this study was to examine the association between plasma hormone concentrations, cervical length, and preterm delivery in twin pregnancies, including the effect of progesterone treatment. MATERIAL AND METHODS: This study included 191 women pregnant with twins from a randomized placebo-controlled trial. A baseline blood sample was collected at 18-24 weeks before treatment with vaginal progesterone (n = 95) or placebo pessaries (n = 96), and 167 (87.4%) women had a second sample collected after 4-8 weeks of treatment. At baseline, 155 (81.2%) women had their cervical length measured. Progesterone, estradiol, and unconjugated estriol concentration was measured, and the association between hormone concentrations, cervical length, and gestational age at delivery was examined. Hormone concentrations were compared in the placebo and progesterone group. Statistical analysis included Spearman's rho, Mann-Whitney U test, Cuzick's test for trends, and linear regression analyses. RESULTS: A short cervical length was associated with preterm delivery. Cervical length and hormone concentrations were not associated (Spearman's rho; progesterone -.05, estradiol .04, estriol .08). Decreasing gestational age at delivery was associated with higher progesterone and estradiol concentrations at baseline (P trend; progesterone 0.04, estradiol 0.02) but not in the second sample or in the weekly change between samples. Progesterone treatment did not increase the progesterone concentration. CONCLUSIONS: Plasma concentrations of progesterone, estradiol, and unconjugated estriol at 18-24 weeks are not associated with cervical length or preterm delivery in twin pregnancies. Vaginal progesterone treatment does not increase the circulating progesterone concentration in twin pregnancies. Cervical length, but not hormone concentration, is predictive of preterm delivery in twin gestations.